Skip to content

skip livemigration for centos#4801

Merged
yadvr merged 2 commits intoapache:4.15from
shapeblue:centosLive
Apr 10, 2021
Merged

skip livemigration for centos#4801
yadvr merged 2 commits intoapache:4.15from
shapeblue:centosLive

Conversation

@DaanHoogland
Copy link
Contributor

@DaanHoogland DaanHoogland commented Mar 12, 2021

Description

This PR skips live migration on centos as the default installations for qemu/libvirt don't support it.

Fixes #4757

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

@DaanHoogland
Copy link
Contributor Author

@weizhouapache @shwstppr @rhtyd please let me know if this needs to go on 4.15. as it is only an intergration test skipping, I didn't deeem it important.

@DaanHoogland DaanHoogland mentioned this pull request Mar 12, 2021
12 tasks
@shwstppr
Copy link
Contributor

@DaanHoogland there seems some issue with one of the test - test_02_migrate_VM_with_two_data_disks. I found it failing on Ubuntu18 as well,

[root@pr4710-t97-kvm-ubuntu18-marvin marvin]# nosetests --with-xunit --xunit-file=results.xml --with-marvin --marvin-config=./pr4710-t97-kvm-ubuntu18-advanced-cfg -s -a tags=advanced --hypervisor=KVM tests/smoke/atest.py

==== Marvin Init Started ====

=== Marvin Parse Config Successful ===

=== Marvin Setting TestData Successful===

==== Log Folder Path: /marvin/MarvinLogs/Mar_12_2021_08_37_07_133UM3. All logs will be available here ====

=== Marvin Init Logging Successful===

==== Marvin Init Successful ====
=== TestName: test_01_migrate_VM_and_root_volume | Status : SUCCESS ===

=== TestName: test_02_migrate_VM_with_two_data_disks | Status : EXCEPTION ===

2021-03-12 08:39:14,524 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-73:ctx-ad649327 job-188/job-189 ctx-896c5319) (logid:3d8a7bb9) Failed to copy volume
com.cloud.utils.exception.CloudRuntimeException: Copy operation failed in 'StorageSystemDataMotionStrategy.copyAsync': null
	at org.apache.cloudstack.storage.motion.StorageSystemDataMotionStrategy.copyAsync(StorageSystemDataMotionStrategy.java:1901)
	at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:119)
	at org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:1645)
	at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1181)
	at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:3000)
	at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:5524)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
	at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5631)
	at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
2021-03-12 08:39:14,524 DEBUG [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-73:ctx-ad649327 job-188/job-189 ctx-896c5319) (logid:3d8a7bb9) Failed to migrated vm VM[User|i-9-20-VM] along with its volumes. com.cloud.utils.exception.CloudRuntimeException: Copy operation failed in 'StorageSystemDataMotionStrategy.copyAsync': null
2021-03-12 08:39:14,524 INFO  [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-73:ctx-ad649327 job-188/job-189 ctx-896c5319) (logid:3d8a7bb9) Migration was unsuccessful.  Cleaning up: VM[User|i-9-20-VM]
2021-03-12 08:39:14,524 WARN  [c.c.a.AlertManagerImpl] (Work-Job-Executor-73:ctx-ad649327 job-188/job-189 ctx-896c5319) (logid:3d8a7bb9) AlertType:: 17 | dataCenterId:: 1 | podId:: 1 | clusterId:: null | message:: Unable to migrate vm i-9-20-VM from host template-ubuntu180401-2 in zone pr4710-t97-kvm-ubuntu18 and pod pr4710-t97-kvm-ubuntu18
2021-03-12 08:39:14,525 DEBUG [c.c.a.AlertManagerImpl] (Work-Job-Executor-73:ctx-ad649327 job-188/job-189 ctx-896c5319) (logid:3d8a7bb9) Have already sent: 1 emails for alert type '17' -- skipping send email
2021-03-12 08:39:14,526 DEBUG [c.c.a.m.AgentAttache] (Work-Job-Executor-73:ctx-ad649327 job-188/job-189 ctx-896c5319) (logid:3d8a7bb9) Seq 1-6154731840755204180: Routed from 32985684378381
2021-03-12 08:39:14,527 DEBUG [c.c.a.t.Request] (Work-Job-Executor-73:ctx-ad649327 job-188/job-189 ctx-896c5319) (logid:3d8a7bb9) Seq 1-6154731840755204180: Sending  { Cmd , MgmtId: 32985684378381, via: 1(template-ubuntu180401), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","forceStop":"false","volumesToDisconnect":[],"vmName":"i-9-20-VM","executeInSequence":"false","wait":"0"}}] }
2021-03-12 08:39:14,530 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-73:ctx-ad649327 job-188/job-189 ctx-896c5319) (logid:3d8a7bb9) VM state transitted from :Migrating to Running with event: OperationFailedvm's original host id: 2 new host id: 2 host id before state transition: 1
2021-03-12 08:39:14,534 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-73:ctx-ad649327 job-188/job-189 ctx-896c5319) (logid:3d8a7bb9) Hosts's actual total CPU: 6300 and CPU after applying overprovisioning: 12600
2021-03-12 08:39:14,534 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-73:ctx-ad649327 job-188/job-189 ctx-896c5319) (logid:3d8a7bb9) Hosts's actual total RAM: (6.77 GB) 7266349056 and RAM after applying overprovisioning: (6.77 GB) 7266349056
2021-03-12 08:39:14,534 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-73:ctx-ad649327 job-188/job-189 ctx-896c5319) (logid:3d8a7bb9) release cpu from host: 1, old used: 1200,reserved: 0, actual total: 6300, total with overprovisioning: 12600; new used: 1100,reserved:0; movedfromreserved: false,moveToReserveredfalse
2021-03-12 08:39:14,534 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-73:ctx-ad649327 job-188/job-189 ctx-896c5319) (logid:3d8a7bb9) release mem from host: 1, old used: (1.75 GB) 1879048192,reserved: (0 bytes) 0, total: (6.77 GB) 7266349056; new used: (1.50 GB) 1610612736,reserved:(0 bytes) 0; movedfromreserved: false,moveToReserveredfalse
2021-03-12 08:39:14,539 DEBUG [c.c.n.NetworkModelImpl] (Work-Job-Executor-73:ctx-ad649327 job-188/job-189 ctx-896c5319) (logid:3d8a7bb9) Service SecurityGroup is not supported in the network id=209
2021-03-12 08:39:14,548 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-73:ctx-ad649327 job-188/job-189 ctx-896c5319) (logid:3d8a7bb9) Invocation exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-9-20-VM] along with its volumes. 
2021-03-12 08:39:14,548 INFO  [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-73:ctx-ad649327 job-188/job-189 ctx-896c5319) (logid:3d8a7bb9) Rethrow exception com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-9-20-VM] along with its volumes. 
2021-03-12 08:39:14,548 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-73:ctx-ad649327 job-188/job-189) (logid:3d8a7bb9) Done with run of VM work job: com.cloud.vm.VmWorkMigrateWithStorage for VM 20, job origin: 188
2021-03-12 08:39:14,548 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-73:ctx-ad649327 job-188/job-189) (logid:3d8a7bb9) Unable to complete AsyncJobVO {id:189, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkMigrateWithStorage, cmdInfo: rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXMCAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdXRpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZEoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAAAIAAAAAAAAAAgAAAAAAAAAUdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAAAAAAAAAABAAAAAAAAAAJzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYWRGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAMdwgAAAAQAAAAAnNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAAI3NxAH4ACAAAAAAAAAACc3EAfgAIAAAAAAAAACRxAH4AC3g, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 32985684378381, completeMsid: null, lastUpdated: null, lastPolled: null, created: Fri Mar 12 08:39:13 UTC 2021, removed: null}, job origin:188
com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-9-20-VM] along with its volumes. 
	at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1186)
	at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:3000)
	at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:5524)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
	at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5631)
	at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
2021-03-12 08:39:14,554 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-73:ctx-ad649327 job-188/job-189) (logid:3d8a7bb9) Complete async job-189, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AEFGYWlsZWQgdG8gbWlncmF0ZWQgdm0gVk1bVXNlcnxpLTktMjAtVk1dIGFsb25nIHdpdGggaXRzIHZvbHVtZXMuIHVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAFnNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAEonQAA2FwcHQAPW9yZy5hcGFjaGUuY2xvdWRzdGFjay5lbmdpbmUub3JjaGVzdHJhdGlvbi5Wb2x1bWVPcmNoZXN0cmF0b3J0ABdWb2x1bWVPcmNoZXN0cmF0b3IuamF2YXQADm1pZ3JhdGVWb2x1bWVzcHBzcQB-AAwBAAALuHEAfgAOdAAmY29tLmNsb3VkLnZtLlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGx0AB5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsLmphdmF0AB1vcmNoZXN0cmF0ZU1pZ3JhdGVXaXRoU3RvcmFnZXBwc3EAfgAMAQAAFZRxAH4ADnEAfgATcQB-ABRxAH4AFXBwc3EAfgAMAv____5wdAAtamRrLmludGVybmFsLnJlZmxlY3QuTmF0aXZlTWV0aG9kQWNjZXNzb3JJbXBsdAAdTmF0aXZlTWV0aG9kQWNjZXNzb3JJbXBsLmphdmF0AAdpbnZva2UwdAAJamF2YS5iYXNldAAHMTEuMC4xMHNxAH4ADAIAAAA-cHEAfgAYcQB-ABl0AAZpbnZva2VxAH4AG3EAfgAcc3EAfgAMAgAAACtwdAAxamRrLmludGVybmFsLnJlZmxlY3QuRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbHQAIURlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGwuamF2YXEAfgAecQB-ABtxAH4AHHNxAH4ADAIAAAI2cHQAGGphdmEubGFuZy5yZWZsZWN0Lk1ldGhvZHQAC01ldGhvZC5qYXZhcQB-AB5xAH4AG3EAfgAcc3EAfgAMAQAAAGtxAH4ADnQAImNvbS5jbG91ZC52bS5WbVdvcmtKb2JIYW5kbGVyUHJveHl0ABpWbVdvcmtKb2JIYW5kbGVyUHJveHkuamF2YXQAD2hhbmRsZVZtV29ya0pvYnBwc3EAfgAMAQAAFf9xAH4ADnEAfgATcQB-ABRxAH4AKHBwc3EAfgAMAQAAAGZxAH4ADnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9icHBzcQB-AAwBAAACbHEAfgAOdAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNKb2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0cHBzcQB-AAwBAAAAMHEAfgAOdAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ29udGV4dFJ1bm5hYmxlLmphdmF0AANydW5wcHNxAH4ADAEAAAA3cQB-AA50AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhbGxwcHNxAH4ADAEAAABmcQB-AA50AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0cQB-ADh0AA9jYWxsV2l0aENvbnRleHRwcHNxAH4ADAEAAAA0cQB-AA5xAH4AO3EAfgA4dAAOcnVuV2l0aENvbnRleHRwcHNxAH4ADAEAAAAtcQB-AA50ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4ANHEAfgA1cHBzcQB-AAwBAAACOHEAfgAOcQB-AC9xAH4AMHEAfgA1cHBzcQB-AAwCAAACA3B0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AOXEAfgAbcQB-ABxzcQB-AAwCAAABCHB0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-ADVxAH4AG3EAfgAcc3EAfgAMAgAABGhwdAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJxAH4AG3EAfgAcc3EAfgAMAgAAAnRwdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgBKcQB-ADVxAH4AG3EAfgAcc3EAfgAMAgAAA0JwdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ADVxAH4AG3EAfgAcc3IAH2phdmEudXRpbC5Db2xsZWN0aW9ucyRFbXB0eUxpc3R6uBe0PKee3gIAAHhweAAAEJp3CAAAAAAAAAAAeA
2021-03-12 08:39:14,555 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-73:ctx-ad649327 job-188/job-189) (logid:3d8a7bb9) Publish async job-189 complete on message bus
2021-03-12 08:39:14,555 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-73:ctx-ad649327 job-188/job-189) (logid:3d8a7bb9) Wake up jobs related to job-189
2021-03-12 08:39:14,555 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-73:ctx-ad649327 job-188/job-189) (logid:3d8a7bb9) Update db status for job-189
2021-03-12 08:39:14,560 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-73:ctx-ad649327 job-188/job-189) (logid:3d8a7bb9) Wake up jobs joined with job-189 and disjoin all subjobs created from job- 189
2021-03-12 08:39:14,564 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-73:ctx-ad649327 job-188/job-189) (logid:3d8a7bb9) Done executing com.cloud.vm.VmWorkMigrateWithStorage for job-189
2021-03-12 08:39:14,565 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-73:ctx-ad649327 job-188/job-189) (logid:3d8a7bb9) Remove job-189 from job monitoring
2021-03-12 08:39:14,575 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-109:ctx-3fa14f5c job-188) (logid:3d8a7bb9) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd
com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-9-20-VM] along with its volumes. 
	at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1186)
	at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:3000)
	at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:5524)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
	at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5631)
	at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

I'll check if I can fix that

@weizhouapache
Copy link
Member

@weizhouapache @shwstppr @rhtyd please let me know if this needs to go on 4.15. as it is only an intergration test skipping, I didn't deeem it important.

@DaanHoogland To be honest, I suspect the failures are caused by some changes in code, not infrastructure/os side.

@weizhouapache
Copy link
Member

@DaanHoogland I checked the result of pr #4582 which are good.

Test Deploy Virtual Machine ... === TestName: test_01_deploy_vm | Status : SUCCESS ===
ok
Test advanced zone virtual router ... === TestName: test_02_advZoneVirtualRouter | Status : SUCCESS ===
ok
Test Multiple Deploy Virtual Machine ... === TestName: test_04_deploy_vm_multiple | Status : SUCCESS ===
ok
Test Stop Virtual Machine ... === TestName: test_05_stop_vm | Status : SUCCESS ===
ok
Test Force Stop Virtual Machine ... === TestName: test_06_stop_vm_forced | Status : SUCCESS ===
ok
Test Start Virtual Machine ... === TestName: test_07_start_vm | Status : SUCCESS ===
ok
Test Reboot Virtual Machine ... === TestName: test_08_reboot_vm | Status : SUCCESS ===
ok
Test destroy Virtual Machine ... === TestName: test_09_destroy_vm | Status : SUCCESS ===
ok
Test recover Virtual Machine ... === TestName: test_10_restore_vm | Status : SUCCESS ===
ok
Test migrate VM ... === TestName: test_11_migrate_vm | Status : SUCCESS ===
ok
Test destroy(expunge) Virtual Machine ... === TestName: test_12_expunge_vm | Status : SUCCESS ===
ok
Test for attach and detach ISO to virtual machine ... === TestName: test_13_attachAndDetach_iso | Status : SUCCESS ===
ok
Test secure VM migration ... === TestName: test_14_secure_to_secure_vm_migration | Status : SUCCESS ===
ok
Test destroy Virtual Machine ... === TestName: test_15_secured_to_nonsecured_vm_migration | Status : SUCCESS ===
ok
Test Non-secured VM Migration ... === TestName: test_16_nonsecured_to_secured_vm_migration | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 15 tests in 1172.225s

OK

Strange I cannot find some test cases in test/integration/smoke/test_vm_life_cycle.py and its history !!

the test cases in result for pr #4531 are same as what we see in test/integration/smoke/test_vm_life_cycle.py

Test advanced zone virtual router ... === TestName: test_advZoneVirtualRouter | Status : SUCCESS ===
ok
Test Deploy Virtual Machine ... === TestName: test_deploy_vm | Status : SUCCESS ===
ok
Test Multiple Deploy Virtual Machine ... === TestName: test_deploy_vm_multiple | Status : SUCCESS ===
ok
Test VM will be migrated with it's root volume ... === TestName: test_01_migrate_VM_and_root_volume | Status : EXCEPTION ===
ERROR
Test VM will be migrated with it's root volume ... === TestName: test_02_migrate_VM_with_two_data_disks | Status : EXCEPTION ===
ERROR
Test VM will be migrated with it's root volume ... SKIP: VM Migration with Volumes is not supported on other than VMware
Test VM will be migrated with it's root volume ... SKIP: VM Migration with Volumes is not supported on other than VMware
Test VM will be migrated with it's root volume ... SKIP: VM Migration with Volumes is not supported on other than VMware
Test secure VM migration ... === TestName: test_01_secure_vm_migration | Status : SUCCESS ===
ok
Test Non-secured VM Migration ... === TestName: test_02_unsecure_vm_migration | Status : SUCCESS ===
ok
Test destroy Virtual Machine ... === TestName: test_03_secured_to_nonsecured_vm_migration | Status : SUCCESS ===
ok
Test Non-secured VM Migration ... === TestName: test_04_nonsecured_to_secured_vm_migration | Status : SUCCESS ===
ok
Test the following: ... SKIP: Skipping test: Reason -  hypervisorNotSupported
Test the following for all found ovf templates: ... SKIP: Skipping test: Reason -  hypervisorNotSupported
Test Stop Virtual Machine ... === TestName: test_01_stop_vm | Status : SUCCESS ===
ok
Test Force Stop Virtual Machine ... === TestName: test_01_stop_vm_forced | Status : SUCCESS ===
ok
Test Start Virtual Machine ... === TestName: test_02_start_vm | Status : SUCCESS ===
ok
Test Reboot Virtual Machine ... === TestName: test_03_reboot_vm | Status : SUCCESS ===
ok
Test destroy Virtual Machine ... === TestName: test_06_destroy_vm | Status : SUCCESS ===
ok
Test recover Virtual Machine ... === TestName: test_07_restore_vm | Status : SUCCESS ===
ok
Test migrate VM ... === TestName: test_08_migrate_vm | Status : SUCCESS ===
ok
Test destroy(expunge) Virtual Machine ... === TestName: test_09_expunge_vm | Status : SUCCESS ===
ok
Test for attach and detach ISO to virtual machine ... === TestName: test_10_attachAndDetach_iso | Status : SUCCESS ===
ok
Test destroy Virtual Machine and it's volumes ... === TestName: test_11_destroy_vm_and_volumes | Status : SUCCESS ===
ok

self.skipTest("Requires at least two hosts for performing migration related tests")


for host in self.hosts:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DaanHoogland it seems not be a CentOS issue, but issue with test scripts or libvrt/kvm plugin.
let's skip the live migration with volume on kvm at first, so trillian test for other prs is good.

we can fix the issue afterwards. it needs investigation.

could you change the lines to skip the test by checking the host hypervisor instead of host os ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could, but since the tests are for KVM only anyway, that would effectively mean disabling the tests completely.

@GabrielBrascher GabrielBrascher self-requested a review March 12, 2021 12:52
@yadvr yadvr changed the base branch from master to 4.15 April 1, 2021 07:29
@yadvr yadvr changed the base branch from 4.15 to master April 1, 2021 07:30
@yadvr
Copy link
Member

yadvr commented Apr 1, 2021

@DaanHoogland can you change the base to 4.15? The issue also affects PRs targeting 4.15 branch

@DaanHoogland DaanHoogland changed the base branch from master to 4.15 April 2, 2021 12:13
@DaanHoogland DaanHoogland modified the milestones: 4.16.0.0, 4.15.1.0 Apr 2, 2021
@blueorangutan
Copy link

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 311

@DaanHoogland DaanHoogland reopened this Apr 2, 2021
@blueorangutan
Copy link

Trillian test result (tid-335)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 33431 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4801-t335-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Smoke tests completed. 83 look OK, 3 have error(s)
Only failed tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestKubernetesCluster>:teardown Error 73.95 test_kubernetes_clusters.py
runTest Error 0.00 test_vm_life_cycle.py
test_01_redundant_vpc_site2site_vpn Failure 279.28 test_vpc_vpn.py

@yadvr yadvr closed this Apr 5, 2021
@yadvr yadvr reopened this Apr 5, 2021
@yadvr
Copy link
Member

yadvr commented Apr 5, 2021

@blueorangutan package

@blueorangutan
Copy link

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 319

@yadvr
Copy link
Member

yadvr commented Apr 5, 2021

@blueorangutan test

@blueorangutan
Copy link

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@blueorangutan
Copy link

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 339

@yadvr
Copy link
Member

yadvr commented Apr 6, 2021

@blueorangutan test

@blueorangutan
Copy link

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@yadvr yadvr changed the title skip lifemigration for centos skip livemigration for centos Apr 6, 2021
@blueorangutan
Copy link

Trillian test result (tid-364)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 35044 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4801-t364-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Smoke tests completed. 85 look OK, 2 have error(s)
Only failed tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestKubernetesCluster>:teardown Error 80.66 test_kubernetes_clusters.py
runTest Error 0.00 test_vm_life_cycle.py

@yadvr
Copy link
Member

yadvr commented Apr 7, 2021

@DaanHoogland looks like a runtTest (test initialisation/setup) failure, can you check?

@blueorangutan
Copy link

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 343

@DaanHoogland
Copy link
Contributor Author

@rhtyd a py3 print statement was in the way. taken it out.

@blueorangutan
Copy link

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 348

@yadvr
Copy link
Member

yadvr commented Apr 7, 2021

@blueorangutan test

@blueorangutan
Copy link

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@davidjumani
Copy link
Contributor

@blueorangutan test centos8 kvm-centos8

@blueorangutan
Copy link

@davidjumani a Trillian-Jenkins test job (centos8 mgmt + kvm-centos8) has been kicked to run smoke tests

@davidjumani
Copy link
Contributor

@blueorangutan test ubuntu18 kvm-ubuntu18

@blueorangutan
Copy link

@davidjumani unsupported parameters provided. Supported mgmt server os are: centos6, centos7, centos8, ubuntu. Supported hypervisors are: kvm-centos6, kvm-centos7, kvm-centos8, kvm-ubuntu, xenserver-71, xenserver-65sp1, vmware-67u3, vmware-65u2, vmware-60u2, vmware-55u3, xcpng76, xcpng80, xcpng81, xenserver-74, xcpng74

@davidjumani
Copy link
Contributor

@blueorangutan test ubuntu18 kvm-ubuntu18

@blueorangutan
Copy link

@davidjumani a Trillian-Jenkins test job (ubuntu18 mgmt + kvm-ubuntu18) has been kicked to run smoke tests

@blueorangutan
Copy link

Trillian test result (tid-375)
Environment: kvm-centos8 (x2), Advanced Networking with Mgmt server 8
Total time taken: 36648 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4801-t375-kvm-centos8.zip
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vm_iso_uefi.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_outofbandmanagement.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 83 look OK, 4 have error(s)
Only failed tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestDeployVMFromISOWithUefi>:setup Error 0.00 test_deploy_vm_iso_uefi.py
ContextSuite context=TestKubernetesCluster>:teardown Error 93.30 test_kubernetes_clusters.py
test_oobm_background_powerstate_sync Failure 21.84 test_outofbandmanagement.py
test_oobm_enabledisable_across_clusterzones Error 35.23 test_outofbandmanagement.py
test_oobm_issue_power_cycle Error 19.77 test_outofbandmanagement.py
test_oobm_issue_power_off Error 20.57 test_outofbandmanagement.py
test_oobm_issue_power_on Error 19.75 test_outofbandmanagement.py
test_oobm_issue_power_reset Error 19.74 test_outofbandmanagement.py
test_oobm_issue_power_soft Error 22.27 test_outofbandmanagement.py
test_oobm_issue_power_status Error 20.97 test_outofbandmanagement.py
test_oobm_multiple_mgmt_server_ownership Failure 27.88 test_outofbandmanagement.py
test_oobm_zchange_password Error 7.38 test_outofbandmanagement.py
test_hostha_kvm_host_degraded Error 8.89 test_hostha_kvm.py
test_hostha_kvm_host_fencing Error 11.26 test_hostha_kvm.py
test_hostha_kvm_host_recovering Error 8.80 test_hostha_kvm.py

@blueorangutan
Copy link

Trillian test result (tid-376)
Environment: kvm-ubuntu18 (x2), Advanced Networking with Mgmt server u18
Total time taken: 36869 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4801-t376-kvm-ubuntu18.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_reset_vm_on_reboot.py
Intermittent failure detected: /marvin/tests/smoke/test_resource_accounting.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dns.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dnsservice.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_iptables_default_policy.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_routers.py
Intermittent failure detected: /marvin/tests/smoke/test_service_offerings.py
Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_host_maintenance.py
Smoke tests completed. 75 look OK, 12 have error(s)
Only failed tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestKubernetesCluster>:teardown Error 80.41 test_kubernetes_clusters.py
ContextSuite context=TestResetVmOnReboot>:setup Error 0.00 test_reset_vm_on_reboot.py
ContextSuite context=TestRAMCPUResourceAccounting>:setup Error 0.00 test_resource_accounting.py
ContextSuite context=TestRouterDHCPHosts>:setup Error 0.00 test_router_dhcphosts.py
ContextSuite context=TestRouterDHCPOpts>:setup Error 0.00 test_router_dhcphosts.py
ContextSuite context=TestRouterDns>:setup Error 0.00 test_router_dns.py
ContextSuite context=TestRouterDnsService>:setup Error 0.00 test_router_dnsservice.py
ContextSuite context=TestRouterIpTablesPolicies>:setup Error 0.00 test_routers_iptables_default_policy.py
ContextSuite context=TestVPCIpTablesPolicies>:setup Error 0.00 test_routers_iptables_default_policy.py
test_01_isolate_network_FW_PF_default_routes_egress_true Error 0.25 test_routers_network_ops.py
test_02_isolate_network_FW_PF_default_routes_egress_false Error 0.29 test_routers_network_ops.py
ContextSuite context=TestRedundantIsolateNetworks>:setup Error 1.49 test_routers_network_ops.py
ContextSuite context=TestRouterServices>:setup Error 0.00 test_routers.py
ContextSuite context=TestCpuCapServiceOfferings>:setup Error 0.00 test_service_offerings.py
ContextSuite context=TestServiceOfferings>:setup Error 0.15 test_service_offerings.py
ContextSuite context=TestSnapshotRootDisk>:setup Error 0.00 test_snapshots.py
test_01_migrate_VM_and_root_volume Error 70.75 test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks Error 54.70 test_vm_life_cycle.py
test_01_secure_vm_migration Error 109.28 test_vm_life_cycle.py
test_02_unsecure_vm_migration Error 169.94 test_vm_life_cycle.py
test_08_migrate_vm Error 45.95 test_vm_life_cycle.py

@blueorangutan
Copy link

Trillian test result (tid-374)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40211 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4801-t374-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_iso.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_metrics_api.py
Intermittent failure detected: /marvin/tests/smoke/test_projects.py
Smoke tests completed. 86 look OK, 1 have error(s)
Only failed tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestKubernetesCluster>:teardown Error 81.62 test_kubernetes_clusters.py

@yadvr
Copy link
Member

yadvr commented Apr 9, 2021

@DaanHoogland @davidjumani cc @andrijapanicsb @alexandremattioli - should we consider switching to qemu-kvm-ev in our environment and update cloudstack docs if the default CentOS7 packages don't work? See:
https://lists.centos.org/pipermail/centos-virt/2015-October/004717.html
https://wiki.centos.org/SpecialInterestGroup/Virtualization

Otherwise, I propose we check if these test fail with CentOS8 too? If not, let make the skip/check code more granular to check CentOS distro and version, based on which we should skip the test(s).

@DaanHoogland
Copy link
Contributor Author

@rhtyd I'd like to agree, but having alternative packages for qemu/libvirt would mean that we either cannot install in an isolated environment or we have to package that code with ACS. Neither has my preference. I'd say document it and let it slide/known issue.

@andrijapanicsb
Copy link
Contributor

No, we want to keep the old behaviour of SKIPPING these, David knows the root cause why these are now run, by mistake (previously were "overriden" not to run).

No custom packages pls.

@yadvr
Copy link
Member

yadvr commented Apr 10, 2021

Merging this based on last test run on centos7, tests do get skipped for CentOS7; and Wei's testing as well as my lgtm.
As per original commit 0fbf500 the tests will fail on centos7 without qemu-kvm-ev repo.

@yadvr
Copy link
Member

yadvr commented Apr 10, 2021

@davidjumani for any further fixes for Ubuntu or Trillian, submit separate PRs, thanks.

@yadvr yadvr merged commit 03c05bc into apache:4.15 Apr 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants